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FEDERAL SUPPORT STATEMENT 



The present application is based on U.S. provisional patent application No. 
60/201,581 filed on May 3, 2000, and claim priority to that application. 

BACKGROUND OF THE INVENTION 

1. FIELD OF THE INVENTION 

The present invention relates to the field of image recognition, and in particular to 
a method and apparatus for accelerated handwritten symbol recognition in a pen based 
tablet computer. 

2. BACKGROUND ART 

In some computer systems, handwritten symbols are input to the system. These 
symbols are translated by the computer system to machine readable characters. This 
translation is typically computation intensive. In some computer systems, battery 
operated portable devices for example, the general purpose central processing unit (CPU) 
used for the translation is inefficient in its power consumption during the translation 
operation. Thus, the battery is drained more rapidly. Additionally, some battery operated 
systems are Umited in computational power. When a real-time translation requirement is 
placed on symbol translation, the limited computational power results in a limited degree 
of accuracy in the translation process. These problems can be better understood with a 
review of handwritten data entry. 
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Handwritten Data Entry 



A typical computer system consists of a central processing imit (CPU), main 
5 memory such as random access memory (RAM), a data entry device, including a 
positioning device, a mass storage device such as one or more disk drives, a display 
and/or a printer. In the prior art, the data entry device often consists of a keyboard, on 
which a user enters data by typing. The positioning device of a prior art computer system 
may consist of a "mouse" or other cursor positioning device. 

10 

Computer systems also exist that are directed to handwritten data entry rather than 
keyboard data entry. These systems are often characterized by the use of a pen, stylus, or 
other writing device, to enter handwritten data directly on the display of the computer 
system. Alternatively, these systems may provide for a user to enter data on a digitizing 

15 tablet or other input device, with the image of the written input displayed on a separate 
computer display output device. The writing device for entering handwritten or freestyle 
stroke input information is not limited to a pen or stylus, but may be any input device 
such as a mouse, trackball, pointer, or even a person's fingers. Such systems are not 
necessarily Hmited to receiving data generated by human users. For example, machine 

20 generated data may also be inputted and accepted to such systems. 



One class of this handwriting entry computer system that receives handwritten 
data input is referred to as a "pen based" computer system. Li a pen based computer 
system, a writer can input information on a display by "writing" directly on the display. A 
25 writing device, such as a pen or stylus, is used to enter information on the display, hi a 
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typical pen-based computer system, a user touches the stylus to the display and writes as 
the user would on a piece of paper, by making a series of pen strokes to form letters and 
words. A hne appears on the display that follows the path of travel of the pen point, so 
that the pen strokes appear on the display as ink would appear on a handwritten page. 
5 Thus, the user can enter information into the computer by writing on the display. Pen 
based computers typically have a display surface that serves as both an input receiving 
device and as an output display device. 

Handwritten Data Translation 

10 

One characteristic of handwriting entry computer systems is the ability to translate 
original handwritten symbols into machine readable words or characters for display. This 
translation is accomplished via a "character recognition" algorithm. The handwritten 
symbols are translated into, for example, ASCII characters. After the translation, the 
1 5 appearance of the displayed characters is as if they had been typed in via a keyboard. 

To translate a handwritten character into a machine readable character, the 
handwritten character is compared to a library of characters to determine if there is a 
match. A description, or "template" for each character is defined and stored in memory. 

20 Handwritten characters are compared to the stored templates. Match coefficients, 

reflecting how closely a handwritten character matches the template of a stored character, 
are calculated for each template character. The template character with the highest match 
coefficient is identified. The character represented by this template provides the "best fit" 
for the handwritten character. If the match coefficient for the "best fit" character exceeds 

25 a predetermined minimum threshold, the "best fit" character is adopted. If the match 
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coefficient for the "best fit" character is less than the minimum threshold value, no 
translation is done. If the handwritten character cannot be translated, the character must 
be re-entered. 

A disadvantage of current character recognition algorithms is limited accuracy. 
Often, handwritten characters are not translated at all or are mistranslated as an ASCII 
character other than the handwritten character. The mistranslated character must then be 
rewritten by the user, sometimes repeatedly, until a correct translation is made. 

Handwriting Recognition in Portable Svstems 

A portable pen-based computer systems is constrained by the amount of power 
stored in its battery. Typically, portable pen-based computer systems, which require 
handwriting recognition (HWR), rely on grid based single character recognition, which 
forces users to print characters in stylized formats. This approach is not suitable for 
entering large text segments. A better approach for entering large text segments is to 
enable users to write naturally on the screen in their own personal, unconstructed style 
using HWR algorithms. However, HWR algorithms require a large amount of 
computation to translate handwritten symbols into machine readable characters. Typical 
portable pen-based computer systems lack the computational power necessary to 
satisfactorily perform translations. 

Typical portable pen-based computer systems use a general purpose CPU for 
HWR calculations. Typically, a general purpose CPU is inefficient in power 
consumption during HWR calculations. The general purpose CPU is designed to perform 
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more than HWR calculations, so some functions of the CPU are powered, but not used 
for the HWR calculation. Additionally, a general purpose CPU is inefficient in speed 
during HWR calculations. The general purpose CPU must be able to perform certain 
operating system tasks while completing HWR calculations. Thus, the speed with which 
5 HWR calculations are completed is diminished. As a result, fewer HWR calculations 
may be completed in a hmited amount of time. Thus, if the time for HWR is hmited, the 
accuracy of the translation is also limited. 

Single Symbol Translation 

10 

Typically, portable pen-based computer systems translate one character at a time. 
However, such a scheme is difficult when a user has poor handwriting. For example. 
Figure 1 illustrates a word where single symbol translation is difficult. A single symbol 
translation system has difficulty translating the letters "m" (100), "i" (110), and "n" (120) 
15 in the word "jumping" (130) in Figure L However, the word is legible to the user. 
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SUMMARY OF THE INVENTION 

The present invention provides a naethod and apparatus for accelerated 
handwritten symbol recognition in a pen based tablet computer. In one embodiment, 
5 handwritten symbols are translated into machine readable characters using hidden 

Markov models. In one embodiment, handwritten symbols are translated into machine 
readable characters using special purpose hardware, hi one embodiment, the special 
purpose hardware is a recognition processing unit (RPU) which performs feature 
extraction and recognition. A user inputs the handwritten symbols and software 
10 recognition engine preprocesses the input to a reduced form. In one embodiment, the 
preprocessor is fiilly information preserving. 

The data from the preprocessor is sent to the RPU which performs feature 
extraction and recognition. In one embodiment, the RPU has memory and the RPU 
15 operates on data in its memory. In one embodiment, the RPU uses a hidden Markov 

model (HMM) as a finite state machine that assigns probabilities to a symbol state based 
on the preprocessed data from the handwritten symbol. In another embodiment, the RPU 
recognizes collections of symbols, termed "wordlets," in addition to individual symbols. 

20 In one embodiment, the software recognition engine uses the data from the RPU 

in a postprocessor. The postprocessor computes a stream of symbol observation events 
from data produced by the RPU and writer confirmation data. In one embodiment, the 
postprocessor also uses information about context, spelling, grammar, past word usage 
and user information to improve the accuracy of the symbols produced. 

7 
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BRIEF DESCRIPTION OF THE DRAWINGS 



These and other features, aspects and advantages of the present invention will 
become better understood with regard to the following description, appended claims and 
5 accompanying drawings where: 

Figure 1 is a block diagram of a word where single symbol translation is difficult. 

Figure 2 is a block diagram of a pen-based (tablet) computer handwriting symbol 
10 recognition system in accordance with one embodiment of the present invention. 

Figure 3 is a flow diagram of the process of translating handwritten symbols in 
accordance with one embodiment of the present invention. 

1 5 Figure 4 is a flow diagram of the process of translating handwritten symbols 

wherein the RPU calculates forward and backward probabilities in accordance with one 
embodiment of the present invention. 

Figure 5 is a flow diagram of the process of translating handwritten symbols 
20 wherein the RPU calculates forward probabilities in accordance with one embodiment of 
the present invention. 

Figure 6 is a block diagram of an RPU in accordance with one embodiment of the 
present invention. 

25 
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Figure 7 is a flow diagram of the process of handwritten symbol translation using 
context information in accordance with one embodiment of the present invention. 



Figure 8 is a flow diagram of the process of handwritten symbol translation using 
5 spelling information in accordance with one embodiment of the present invention. 

Figure 9 is a flow diagram of the process of handwritten symbol translation using 
grammar information in accordance with one embodiment of the present invention. 

10 Figure 10 is a flow diagram of the process of handwritten symbol translation using 

past word usage information in accordance with one embodiment of the present 
invention. 

Figure 11 is a flow diagram of the process of handwritten symbol translation using 
1 5 user information in accordance with one embodiment of the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 



The invention is a method and apparatus for accelerated handwritten symbol 
recognition in a pen based tablet computer. In the following description, numerous 
5 specific details are set forth to provide a more thorough description of embodiments of 
the invention. It is apparent, however, to one skilled in the art, that the invention maybe 
practiced without these specific details. In other instances, well known features have not 
been described in detail so as not to obscure the invention. 

10 Handwriting Recognition Calculations 

In one embodiment, HMM calculations are used to determine the probabihty of a 
symbol appearing in a sequence of symbol observations. A HMM with N states, M 
observation symbols, the state alphabet Vs = {Si, S2, . . Sn} and the emission alphabet 
15 Ve = {vi, V2, . . Vm} is defined by the triplet X = (A, B, 71). A is a state transition matrix 
defined as a^ - P(qt+i = Sj | % = Si) for 1 < z < TV and 1 < 7 < A/^ , which means the 

probability that the state at time t+1 is state j given that the state at time t is state i. B is 
the observation probabihty matrix defined as bj(k) = P(vk | qt = Sj) for 1 < y < and 

l<k<M , which means the probability of the observation being Vk given that the state at 
20 time t is state j. p is an initial state distribution defined as rcj = P(qi = Sj), which means 
the probability of the state at time 1 is state i. 

If we have an observation sequence O = (01O2. . .ot), the RPU calculates the 
probabihty of this sequence given the model X. This value is calculated by determining a 
25 series of values termed forward variables defined as ai(t) = P(oi02. . .Ot, qt ^ Si | X), which 
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means the probability of the observation sequence from 1 to t and the state at time t being 
state i given X. These values are calculated by initializing the variable as ai(l) = 7ribi(oi) 
for l<i<N , Further values are calculated using aj(t+l) ^ bi(ot+i) ^ ai(t)aij for 

1 < / < r - 1 and 1 < j <N . The probability of the sequence given X is defined by P(P | 

5 X) = X ai(T). 

1=1 



Similarly, a backward variable is defined as |3i(t) = P(ot+iOt+2. . .Oj | qt = Si, X), 
which means the probabihty of the observation sequence from t+1 to T given state at time 
t being state i given X. The backward variables are initiaHzed as p^T) 1 for 1 < z < TV^ . 

N 

10 Further values are calculated using Pj(t) = ^ aybj(ot+i)Pi(t+l) for 1 < / < T - 1 and 
l<j<N, 



The calculations to compute forward and backward variables are performed in the 
RPU. Thus, probabihties can be calculated for each new symbol to determine which 
15 symbols the new symbol is most likely to be. hi one embodiment, the HMM calculations 
are performed on a general purpose computational unit. 



Pre-processed symbol observations are the input to the HMMs. hi one 
embodiment, the symbol observation alphabet (the emission alphabet) is comprised of 
20 angles of equahzed segments. In other embodiment, more complex symbol observation 
alphabets are used. In one embodiment, at least one HMM is created for each symbol in 
the output alphabet. The probabihty of each symbol given the observation sequence is 
calculated by each symbol's HMM. In one embodiment, a post-processing unit uses the 
information from the HMMs to determine the appropriate symbol. 

11 
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TraininR A, B and n 



It is desirable to select the parameters A, B and nofX that maximize the 
5 probabihty of a sequence in the training set given X, One algorithm used to determine the 
parameters is the Baum- Welch method. The Baum- Welch method guarantees a 
monotonically increasing probability and converges quickly. 



First, a joint event variable is defined as Sij(t) = P(qt ^ Si, qt+i = Sj | 0, 1% which 
10 means that the probability of the state at time t being state i and the state at time t+1 being 
state j given sequence O and X. From the definitions of forward and backward variables, 
this becomes Sij(t) = (ai(t)aijbj(ot+i)pj(t+l)) / P(0 | X), 



Additionally, a state variable is defined as yi(t) = P(qt Si | O, X), which means the 
15 probability of the state at time t being state i given sequence O and X, From the 

definitions of forward and backward variables, this becomes yi(t) = (ai(t)pj(t)) / P(0 | X), 

A new X,X\is calculated as follows. A new a, a', is calculated as a'y = ^ Sij(t) 

T~l T T 

yi(t). Anew b,b', is calculated as b'j= ^ YjCO/^ Yj(t). Anew ti, tx', is 
20 calculated as n\ = Yi(l). 



A variation of the Baum- Welch method, termed the "Levingson method," 
calculates X' as follows when K observation sequences are used to adjust the parameters. 

K r-1 K T-l 

A new a, a', is calculated as a'ij = ^ ^ Sy^^^O / X X! y^^^^^^ ^ 

k=l t=l k=l 
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calculated as b^"^ =2 ^ Yj^^t) / J] |] yj^^^t). A new n, n', is calculated as n'i 

k=l =v^ k=l t=\ 

k=l 

Special Purpose Hardware for Recogtiition Processing 

5 

In one embodiment, handwritten symbols are translated into machine readable 
characters using special purpose hardware. In one embodiment, the special purpose 
hardware is a recognition processing unit (RPU) which performs feature extraction and 
recognition. Li another embodiment, a user inputs the handwritten symbols and software 
10 recognition engine preprocesses the input to a reduced form, hi one embodiment, the 
preprocessor is fiilly information preserving. 

Figure 2 illustrates a pen-based (tablet) computer handwriting symbol recognition 
system in accordance with one embodiment of the present invention. A writer (200) 

15 enters handwritten symbols into the tablet computer (210), The handwritten symbols 
operated upon by a preprocessor (220) running on the main processing unit (MPU). The 
data from the preprocessor is sent to an RPU (230). The RPU is implemented as special- 
purpose hardware. Li one embodiment, the RPU is a circuit configured to perform hidden 
Markov model (HMM) computations. The data from the RPU is used by a postprocessor 

20 (240) running on the MPU to produce an unconfirmed symbol observation (250). 

The unconfirmed symbol observation is presented to the writer. The writer can 
confirm a symbol, reject a symbol or make no determination. The postprocessor uses 
confumed symbol observations (260), rejected symbol observations and unconfirmed 
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symbol observations to adjust how it makes symbol observations. The preprocessor also 
uses confirmed symbol observations and unconfirmed symbol observations to adjust how 
it preprocesses the handwritten symbols. Additionally, training data (270) is used by the 
preprocessor, the RPU, and the postprocessor to adjust their calculation to achieve more 
5 accurate symbol translations. 

The special purpose hardware of the RPU enables the system to calculate more 
handwriting recognition calculations in the same amount of time when compared to a 
system where handwriting recognition calculations are performed by a general purpose 

10 processor (the MPU). In one embodiment, the RPU uses parallel processing to make 
multiple handwriting recognition calculations each clock cycle. Typically, a general 
purpose processor requires multiple clock cycles to perform one handwriting recognition 
calculation, Li one embodiment, the RPU performs eight handwriting recognition 
calculations in parallel for each clock cycle. Since the RPU only performs handwriting 

15 recognition calculations, no power is wasted during the calculation. Thus, the same 
amount of handwriting recognition calculations will require less power when computed 
by an RPU than when computed by a general purpose processor. 

Memory on the RPU 

20 

Li one embodiment, the data from the preprocessor is sent to the RPU which 
performs feature extraction and recognition, hi another embodiment, the RPU has 
memory and the RPU operates on data in its memory. Figure 3 illustrates the process of 
translating handwritten symbols in accordance with one embodiment of the present 
25 invention. At step 300, a user enters handwritten symbols into the system. At step 310, 
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the handwritten symbol data is stored in memory accessible by the MPU. At step 320, a 
preprocessor running on the MPU operates on the handwritten symbols. At step 330, the 
data from the preprocessor is transferred to the memory of an RPU. At step 340, the RPU 
operates on the data in its memory. At step 350, the data from the RPU is transferred to 
5 the memory accessible by the MPU. At step 360, a postprocessor running on the MPU 
generates a symbol observation. 

Figure 4 illustrates the process of translating handwritten symbols wherein the 
RPU calculates forward and backward probabihties in accordance with one embodiment 

10 of the present invention. At step 400, a user enters handwritten symbols into the system. 
At step 410, the handwritten symbol data is stored in memory accessible by the MPU. At 
step 420, a preprocessor running on the MPU operates on the handwritten symbols. At 
step 430, the data from the preprocessor is transferred to the memory of an RPU. At step 
440, the RPU calculates forward and backward probabihties. At step 450, the data from 

1 5 the RPU is transferred to the memory accessible by the MPU. At step 460, a 
postprocessor running on the MPU generates a symbol observation. 

Figure 5 illustrates the process of translating handwritten symbols wherein the 
RPU calculates forward probabilities in accordance with one embodiment of the present 

20 invention. At step 500, a user enters handwritten symbols into the system. At step 505, 
the handwritten symbol data is stored in memory accessible by the MPU. At step 510, a 
preprocessor running on the MPU operates on the handwritten symbols. At step 515, the 
data from the preprocessor is transferred to the memory of an RPU. At step 520, the RPU 
calculates initial forward probabihties, ai(l). At step 525, the RPU calculates successive 

25 forward probabihties (ai(2), ai(3), . . ,, ai(t)) to determine the probabilities of the observed 
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sequences. At step 530, the data from the RPU is transferred to the memory accessible by 
the MPU. At step 535, a postprocessor running on the MPU generates a symbol 
observation. 

5 Figure 6 illustrates an RPU in accordance with one embodiment of the present 

invention. The RPU 600 has a memory unit 610 which stores ay 620, bi(ot+i) 630 and 
values of ai(t) 640 as they are calculated, hiitially, the values of ai(t) are set to their 
initial values. Additionally, the RPU has a HMM calculation unit 650. The HMM has N 
ai(t) units 660. Each ai(t) unit is multiplied by the appropriate ay value in an a^ unit 670. 
10 All of the products are summed in a summation unit 680 and the sum is multipUed by the 
appropriate bi(ot+i) value stored in a bi(ot+i) unit 690. The resulting product is stored in 
the memory unit. Thus, the HMM calculates aj(t+l) = bi(ot+i) ^ ai(t)aij for 1 < ^ < T - 1 

and 1 < 7 < TV . 

15 hi some embodiments, the RPU has multiple HMM calculation units to enable 

multiple HMM calculations to take place in parallel, hi one embodiment, the RPU has N 
HMM calculation units. Thus, values of aj(t+l) are calculated in parallel for all values of 
j' 

20 Symbols and Wordlets 

hi one embodiment, the RPU uses a hidden Markov model (HMM) as a finite 
state machine that assigns probabilities to a symbol based on the preprocessed data from 
the handwritten symbol. For example, a handwritten symbol may have a one in three 
25 probability of being an "e" and a one in four probabiUty of being an "i." hi another 
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embodiment, the RPU recognizes collections of symbols, termed "wordlets," in addition 
to individual symbols. 

For example, the RPU may recognize "tion" or "ing" as one symbol. The output 
5 alphabet contains "tion" and "ing" in addition to "t", "i", "o", "n" and "g". The abihty to 
recognize a handwritten symbol as the wordlet "ing" improves the accuracy of translation. 
For example, in Figure 1, the "i," the "n" and the "g" are all difficult to recognize 
individually. However, the "ing," together, is easier to recognize. Where one letter ends 
and the next begins becomes less important to the determination when the entire wordlet 
1 0 can be recognized by the RPU. 

Probabilistic Context Free Grammar 

Li one embodiment, probabilistic context free grammar information is used to 
1 5 improve the accuracy of symbol translation. A probabilistic context free grammar is 

defined as G = (Vn, Vt, P, S). Vn is a nonterminal feature alphabet defined as Vn =^ {Fi, 
F2, . . Fn} . Vt is a terminal feature alphabet defined as Vt = (wi, W2, . . wm} . AH of 
the production rules of the grammar are of the form Fi FjFk or Fi — > Wk where Fi e Vn 
are nonterminal features and Wk g Vt are terminal features. Fi is set equal to the entire 
20 string of terminals. 

These production rules are defined by tensors A and B as P = (A, B). For the 
nonterminal features, a probabiUty tensor A of rank 3 is defined as ayk = P(Fi FjFk) for 
l<i< N , 1< j < N and l<k< N , For the terminal features, a production probability 
25 matrix Bis defined as bj(k)=P(Fj^Wk) for 1< 7 < and 1<^<M. 
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In one embodiment, the probability of a string of temiinals of length T, W = 
w V^ . .w^, where e Vt is determined given a probabilistic context free grammar 
defined as P(W | G). In one embodiment, the probabihty of a sub-sequence, W^'^ = 
5 w^. . .w"^, termed an "inside probability" is calculated. The inside probability is initiahzed 
as pi(t, t) = bi(w^) for 1 < / < and 1 < ^ < T . Successive inside probabilities are 

N N q~\ 

determined by calculating pi(p, q) = ^ Z Z Pi(P.t)Pk(t+l ,q) for 1 < r < AT . At 

j=\ k=\ t=p 

termination, P(W | G) = Pi(lJ). 



1 0 Similarly, the "outside probability is the probability that the sub-sequence, W^'"^ = 

w^. . .w^, was generated by the nonterminal Fi in the sequence W = w V^. . .w^. The 
outside probability is initialized as ai(l, T) = 6ii for 1 < / < A?^ . Sh = 1 for i = 1 and 5ii ^ 0 
for all other values of i. Successive outside probabilities are determined by calculating 

N N T N N p-l 

ocj(p,q)=2; aykj; ai(p,t)pk(q+l,t)+ "^^T ak(t,q)pi(t,p-l) for 

i=l k=\J^k t=q+l i=l k=\ t=\ 

N 

15 1 < 7 < . At termination, P(W | G) = ^ bj(wVj(t,t) for 1 < r < 7 , 



Training for Probabilistic Context Free Granomar 



A joint feature probabihty is defined as ^ijk(p, q) = 

q-l 

(p^ Z A (p^ OA (t + 1 q) 

20 A? n T\ * ^ parent feature probability is defined as yi(p, q) = 

'^^'r^{\t\'^^ * The joint feature probability and parent feature probability are used to 

Hi (I? ^ ) 

calculate new nonterminal probabiUties and new terminal probabilities. The new 
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T-l T 

nonterminal probabilities are calculated as = Iz^Jzl for 1 < / < A'' and 

p=l q=p 

T 

1 < A: < M . The new terminal probabilities are calculated as bj (k) = ^^^^^'^""^ for 

p=l q=p 

1<J<N 2ndl<k<M , 

5 In one embodiment, the inside probability is used to determine a probability of a 

string of observed terminals given a probabilistic context free grammar. In one 
embodiment, the tensors of the probabihty are trained on a sample language by 
calculating new terminal and nonterminal probabilities using the above equations. In one 
embodiment, the inside probability is calculated using general purpose hardware. In 
10 another embodiment, the inside probability is calculated using special purpose hardware. 

Context Consideration to Improve Accuracy 

In one embodiment, the software recognition engine uses the data from the RPU 
15 in a postprocessor. The postprocessor computes a stream of symbol observation events 
from data produced by the RPU and writer confirmation data. In one embodiment, the 
postprocessor also uses information about context, spelhng, grammar, past word usage 
and user information to improve the accuracy of the symbols produced. 

20 Figure 7 illustrates the process of handwritten symbol translation using context 

information in accordance with one embodiment of the present invention. At step 700, a 
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user enters handwritten symbols into the system. At step 710, a preprocessor running on 
the MPU operates on the handwritten symbols. At step 720, the RPU operates on the data 
in its memory. At step 730, postprocessor running on the MPU operates on the data to 
generate a symbol observation using the context of the handwritten symbol. 

In one embodiment, the postprocessor uses spelling information combined with 
previously generated symbols to determine the current symbol. Figure 8 illustrates the 
process of handwritten symbol translation using spelling information in accordance with 
one embodiment of the present invention. At step 800, a user enters handwritten symbols 
into the system. At step 810, a preprocessor running on the MPU operates on the 
handwritten symbols. At step 820, the RPU operates on the data in its memory. At step 
830, postprocessor ruiming on the MPU adjusts the probability of symbols based on 
whether the symbol, in combination with apphcable previous symbols, would produce a 
correctly spelled word or word segment in the current context. At step 840, the 
postprocessor generates a symbol observation. For example, the postprocessor would 
assign a higher probabiUty to a symbol following a "q" being a '*u" rather than being "ei". 

In another embodiment, the postprocessor uses grammar information combined 
with previously generated symbols to determine the current symbol. Figure 9 illustrates 
the process of handwritten symbol translation using grammar information in accordance 
with one embodiment of the present invention. At step 900, a user enters handwritten 
symbols into the system. At step 910, a preprocessor running on the MPU operates on 
the handwritten symbols. At step 920, the RPU operates on the data in its memory. At 
step 930, postprocessor running on the MPU adjusts the probability of symbols based on 
whether the symbol, in combination with apphcable previous symbols, would produce a 
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set of symbols which are not grammatically incorrectly in the current context. At step 
940, the postprocessor generates a symbol observation. For example, a processor would 
give a higher probabiUty to the symbol following "I am an inventor to" being an "o" than 
being a".". 

5 

In one embodiment, the postprocessor uses past word usage information 
combined with previously generated symbols to determine the current symbol. Figure 10 
illustrates the process of handwritten symbol translation using past word usage 
information in accordance with one embodiment of the present invention. At step 1000, a 

10 user enters handwritten symbols into the system. At step 1010, a preprocessor running on 
the MPU operates on the handwritten symbols. At step 1020, the RPU operates on the 
data in its memory. At step 1030, postprocessor running on the MPU adjusts the 
probabiUty of symbols based on whether the symbol, in combination with apphcable 
previous symbols, would produce a word, or segment of a word, which was previously 

15 input to the system. At step 1040, the postprocessor generates a symbol observation. 

hi one embodiment, the postprocessor uses user information combined with 
previously generated symbols to determine the current symbol. Figure 1 1 illustrates the 
process of handwritten symbol translation using user information in accordance with one 
20 embodiment of the present invention. At step 1 100, a user enters handwritten symbols 
into the system. At step 1 1 10, a preprocessor running on the MPU operates on the 
handwritten symbols. At step 1 120, the RPU operates on the data in its memory. At step 
1 130, postprocessor running on the MPU adjusts the probabiUty of symbols to account 
for known user information. For example, the system may know the user writes "ful" in a 
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consistently distinct manner. At step 1 140, the postprocessor generates a symbol 
observation. 

Thus, a method and apparatus for accelerated handwritten symbol recognition in a 
pen based tablet computer is described in conjunction with one or more specific 
embodiments. The invention is defined by the following claims and their full scope and 
equivalents. 
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